\chapter{Roadmap}

In this appendix, we discuss work to be done in many of the external
libraries that we use in \sysname{}, as well as work to be done to
\sysname{} bootstrapping itself.  There are several reasons for the
existence of this appendix:

\begin{itemize}
\item We want to have a list to return to when we need to prioritize
  what to work on next.
\item We want to give an idea what remains to be done for \sysname{}
  to be a complete \commonlisp{} system.
\item We hope to inspire others to help out by showing tasks to be
  worked on.
\end{itemize}

This appendix is divided into two parts.  The first part discusses
work to be done to external libraries.  Such work is largely
independent of the \sysname{} code base, so can be worked on
independently.  The second part discusses work to be done to the
\sysname{} bootstrap procedure.  This work requires good understanding
of the bootstrapping technique used, and often knowledge about the
details of it.

\section{External libraries}

\subsection{Clostrum}

\subsection{Trucler}

This library provides lexical environments for use by compilers.  It
currently supports its own reference implementation and also SBCL.

It should be possible for a compiler to accomplish all its tasks by
using only Trucler.  In particular, it should not be necessary to
consult the global environment, since the global environment varies
between \commonlisp{} implementations.

To accomplish this goal, Trucler must be augmented with a number of
functionalities, for example to determine whether an expression is a
valid type specifier, and to determine whether a symbol has been
proclaimed \texttt{declaration}.  

One suggestion is this:

\begin{itemize}
\item We add a new function \texttt{describe-declaration} that takes a
  client, and environment and a symbol as arguments, and that returns
  \textit{true} if and only if the symbol has been proclaimed
  \texttt{declaration} in the global environment.
\item We modify the contract of \texttt{add-variable-type} and
  \texttt{add-function-type} so that they are required to return a new
  environment in which the type has been fully expanded.
\end{itemize}

Trucler should also have more backends, at least for CCL, ECL.

\subsection{Khazern}

This library is an extensible implementation of the \texttt{loop}
macro, and is maintained by Tarn W. Burton.  The only possible
improvement I see is the clause parser.  It is written in a simplified
version of combinator parsers, but it doesn't support full
backtracking.  It would probably be better to use an external library.

\subsection{Clostrophilia}

We introduced symbols such as x+1, x-1, and x\^\ to indicate different
``refinement'', but we can eliminate those if we use the new renaming
feature in the bootstrapping procedure.  That way, the Clostrophilia
code looks more ``natural''.  We do need to verify that no two systems
contain both symbol x and one of the variants simultaneously.  We can
then eliminate comments about refinement.

The documentation is embryonic and needs to be improved.

\subsection{Common macros}

Currently, the repository ``Common-macros'' contains two systems,
namely \texttt{common-macros} and \texttt{common-macro-definitions}.
We intended for \texttt{common-macros} to contain standard macros in
the form of AST transformations, but this turned out to be harder than
we had expected, simply because the code for such a transformation
because very ugly.  So we created the system
\texttt{common-macro-definitions} to contain standard macros in the
traditional way, using DEFMACRO forms.  Only
\texttt{common-macro-definitions} is currently used, and
\texttt{common-macros} is incomplete.

We want to remove the system \texttt{common-macros} from this
repository, rename it, and put it in a separate repository, say
\texttt{common-ast-macro-transformations} or something like that.  We
would have liked to rename \texttt{common-macro-definitions} to
\texttt{common-macros}, but it is already used by others, and we don't
want to force them to change their code.

\subsection{Common boot}

The essence of the repository ``Common-boot'' contains methods on
\texttt{architecture.builder-protocol:finish-node}.  Such a method
takes an unfinished abstract syntax tree (AST), i.e., one that
contains unparsed slots, and recursively calls \texttt{convert-ast}
that uses the \texttt{builder} of \texttt{Iconoclast} to parse the
slots so that the result is a finished AST.

But this repository also contains:

\begin{itemize}
\item definitions of node classes of a graph in high-level
  intermediate representation (HIR),
\item code for converting an AST to HIR,
\item code for closure conversion of an AST,
\item an evaluator working on the AST level, and
\item and evaluator working on the HIR level.
\end{itemize}

We think that this repository could be split up into several different
ones, given that some clients will not be interested in all this
functionality.  In particular, we have already ``extracted'' the
definitions of the HIR graph into the repository ``Hirundine''.  The
AST evaluator is no longer used by \sysname{}, but it could be
interesting for some other clients.  The HIR evaluator would be
extracted either to Hirundine or to a separate repository.

Each of the evaluators contains a fairly complete test suite that
should follow the corresponding evaluator to its new repository.

\subsection{Iconoclast}

I would like to introduce slots with a cardinality of zero.  Such a
slot would contain references that are not part of the AST tree
structure, but that may reference other parts of the AST tree.  An
example of this would be the mutual references of a variable
definition and the corresponding references to that variable.  The AST
walker would be modified not to follow objects in slot with
cardinality zero.  Currently, slots that should not be traversed are
handled specially.

Introduce AST types for representing the literals vector.  Perhaps an
AST type \texttt{literals-vector} with no particular value, and that
contains a list of forms to create the global literals vector.
Perhaps a type \texttt{literal-reference} taking with an index into
the literals vector, but no explicit reference to the vector.

Introduce AST types for getting the cells of a global function and the
global value of a variable. These ASTs go in the
\texttt{literals-vector} AST. 

Define transformations for turning existing AST types like
\texttt{literal-ast} into a literals vector.  This is nontrivial
because it requires something similar to \texttt{make-load-form}.

Define transformation for turning \texttt{load-time-value-ast} into an
element of the literals vector.

Define syntax for input/output of ASTs, probably using the syntax
``[class-name :initarg1 value1 ...''.

See section on S-expression syntax for this remark.  Iconoclast will
likely need the ability to represent ASTs for invalid syntax.

\subsection{Consecution}

Create a version of \texttt{remove} (and related functions) that
shares as much structure as possible between the input and the output.

Add documentation strings.

Add compiler macros.  This part is not trivial.  Currently, each
standard function is defined as a generic function with several
methods.  A compiler macro should parse keyword arguments and then
call a function with no keyword parameters.  But then that new
function must be generic and it must be called by the standard
function.

Check that the use of :FROM-END on a list argument uses the technique
in the paper by Durand and Strandh. 

\subsection{Constrictor}

There are a few generic functions defined, but it would probably be
best to get rid of those so that this system can be used without CLOS.

\subsection{Hirundine}

Define instructions for getting cells of global functions and global
values of special variables.

\subsection{Ecclesia}

This library contains two different representations of lambda lists.
One representation uses instances of standard classes with slots for
each parameter type.  The other representation uses ordinary lists,
but with a canonical representation that is then easier to access.  I
used to think that the order of the different parameter types was
important, because the \texttt{\&environment} parameter can appear
anywhere in the lambda list.  However, the standard says that 
the \texttt{\&environment} and \texttt{\&whole} parameters are bound
before any other parameter.  Therefor the order does not matter.

So now I think it is better to use instances of standard classes to
represent lambda lists.  Unfortunately, I encourage Alex Wood to
switch from the first representation to the list representation.  It
is therefore not obvious how to fix this library without breaking
existing client code.

Perhaps the solution is to create an entirely new library and
encourage the use of that library instead of Ecclesia. 

\subsection{Posterior}

This library is still embryonic.  The idea is that it contains
definitions of instructions and data for a traditional control-flow
graph in so-called Medium-level Intermediate Representation (MIR).
This representation represents memory addresses explicitly.  The
library also has backends for a variety of processor architectures.
The instructions are similar to those that can be found on most real
processor architectures, and they operate on virtual registers that
can be translated into real registers for specific backends.  This
library is also meant to contain optimizations that can be performed
at the MIR level.

Specific \commonlisp{} implementations translate programs using the
High-level Intermediate Representation (HIR) into MIR, because this
translation involves representations of objects that are specific to
each \commonlisp{} implementation.  But this library should ideally be
possible to use by other languages than \commonlisp{}.

\subsection{Concrete Syntax Tree}

There is a lambda-list parser that should probably not be in this
library. 

\subsection{S-expression syntax}

This library is not directly part of the \sysname{} project, but it is
used to parse \commonlisp{} S-expressions.

Also, this remark is not directly related to \sysname{} but to the
project of creating an integrated development environment.  For that
project, the library should handle invalid syntax.

\subsection{Salmagundi}

This library is maintained by Tarn W Burton.  It could use
documentation strings and perhaps compiler macros.

\subsection{Buoy}

This library is embryonic.  The purpose is to supply \commonlisp{}
implementations of nontrivial arithmetic functions on floating-point
and complex numbers, like \texttt{sin}, \texttt{cos}, \texttt{atan},
etc.  Comments in the library suggest an existing source of
high-quality functions written in C.  These existing functions can be
translated into \commonlisp{}.

\section{Bootstrapping}

\subsection{Replace cell sharing by renaming}

We used to use the macro \texttt{with-intercepted-function-cells} to
share cells between environments.  We may still need this macro in
order to introduce cells containing host functions, but for other
usages, we now have a better macro
\texttt{with-intercepted-function-names} that make it much more
obvious what functions call functions in other environments.  We
should apply this new macro to various Clostrophilia subsystems to
avoid funny symbol names.

\subsection{Maybe use more than one final environment}

The current idea is to make environment number 4 the ``final''
environment to be used as a model for the one in the final system.
But we have already suggested the use of several environments in the
final system, like a separate environment for each nontrivial
subsystem, like the compiler.  But there might already be some such
nontrivial subsystems that could be put in separate environments.  For
that to be a useful thing to do, the subsystem should contain a
significant amount of code, but supply very few entry points for
general use, like the compiler with \texttt{compile},
\texttt{compile-file}, and \texttt{eval}.  One such existing system
would be \texttt{ctype} that supplies only \texttt{typep} and
\texttt{subtypep}. 

\subsection{Change \texttt{common-boot:closure} into simple functions}
